The Great Designers advocate using the Dick, Carey and Carey model of formative evaluation. Our product is designed to be self-instructional, therefore the emphasis should be on collecting data, analyzing data, and revising instruction based on the analysis of the data. We will employ the following formative evaluations in our evaluation process: peer evaluations, learner evaluations, instructional designer observations, and subject matter expert evaluations.
We asked our instructional design peers to evaluate our Great Depression Book Builder lesson and provide feedback on their experience and impressions through an online survey. Eight students from EDET 722 served as peer evaluators of the design, presentation of information and content of our Book Builder lesson. The survey respondents were consistent and forthcoming in their objective findings, opinions and suggestions.
The rating scale for the fourteen individual prompts that constitute the “The Great Depression” formative evaluation is:
Strongly Agree
Agree
Disagree
Strongly Disagree
All of the input, excepting one, was either Agree or Strongly Agree. The one exception was a “disagree” recorded for the prompt, “The Book Builder lesson makes me want to learn more about the Great Depression.” General responses indicate, “The Great Depression Book Builder site is easy to use. The content of the lesson is written in clear and simple language. The content is easy to understand and follow. The lesson has an attractive presentation I want to see more social studies content presented in this fashion.” This generally positive response to the lesson’s content and delivery method indicates a basis for the continuation of development.
In addition, the peer survey included 1 question which related to the evaluator’s overall impression of the product. Using a scale of Very Good, Good, Average, Poor and Very Poor, respondents answered, “Overall, how do you rate the Great Depression Book Builder lesson?” The Great Designers note with pleasure that 25% rated the Great Depression Book Builder lesson “very good” and 75% rated it “good”.
Two “open-ended” questions offered the opportunity for respondents to identify the components of the lesson that they liked the best as well as the least. Our peers appreciated the audio clips, music, external article(s), glossary and student response section the best. The least liked aspects were related to the “over long” length of the lesson and overabundance of text, lack of audio/narration on certain pages.
The final question requested valued input from our peers regarding suggestions for improvement to our Book Builder lesson. A summation of the suggestions indicated a need to reduce the text significantly on many of our pages, perhaps replacing it with a narrator coach or narration clips. Another peer suggested the inclusion of more of what makes the content interactive; meaning more audio, graphics and video links on every page. The utilization of our Coaches could be improved by programming them to suggest clicking on available audio clips. One reviewer indicated this change would make them more appealing and exciting. And finally was an appeal for the use of more color on the pages in pursuit of a more attractive design.
The three questions that received the lowest rankings were:
The lesson is interesting and engaging (13% strongly approve, 66% approve)
The lesson has an attractive presentation. (13% strongly approve, 66% approve)
The Book Builder lesson makes me want to learn more about the Great Depression. (25% strongly approve, 63% approve, 13% disapprove)
This is particularly troubling because our stated goal is to improve learner scores by increasing learner engagement.
In summation, the input from our peers (external instructional designers) was invaluable. This information in combination with the target student observations, subject matter expert opinions and the experience of our Instructional Design team guided the appropriate modifications to our Book Builder Lesson.
Three high school students completed the Great Depression Book Builder lesson and provided feedback on their experience and impressions. Students completed the lesson individually with an instructional designer in attendance and took the same online survey administered to our peer external instructional designers. The students' responses were aggregated with the peer responses in the results reported above.
Two members of our design team completed observations of a total of three high school students' interaction with the product interface. Observations were recorded related to the level of attention, enthusiasm and reaction to the lesson’s delivery method, content, and student response. Instructional designers used the observation form to complete each observation.
Observation Results
Students spent approximately 1 to 5 minutes on each page. Total time to complete the Book Builder ranged from 30 minutes to 1 hour.
Generally, students did not access all the interactive links. On most pages, only 33% of the students accessed all the interactive links, and students gave the audio and video clips no attention. Students often questioned if they had to watch an entire clip. Students commented that the NPR story was too long and did not listen to the entire story. One student did not even open the story. The students did not use the glossary at all.
Students were inconsistent in launching the coaches. The voice of Hali was difficult to understand without reading along, and students found the coaches to be too slow and too immature for the high school population. Overall the coaches seemed to be a distraction to the students.
Students were also inconsistent in using the student response area. On most pages, 66%-100% of students used the student response area. Students asked if they really had to answer all the questions and expressed frustration that the student response area was small and difficult to edit. On some pages, students asked for help in answering the questions (e.g., Do you know the unemployment rate? What movie stars would I know from the 1930s?).
Overall, students seemed to be at least moderately engaged with the lesson although the slow speed of the coaches and the pages was a distraction. The content was age- and grade-appropriate. The questions were challenging and required the students to put forth effort and thought. On some occasions, students seemed to make connections between the lesson content and prior learning or personal experiences. One student did not seem to put forth much effort and did not access many links. He attributed this lack of effort to not wanting to wait on the school’s slow internet and said that he would access the links if he were on a faster internet connection.
The subject matter expert evaluation sought to answer whether or not the content selected is appropriate given the objectives chosen by the team for the Great Depression Book Builder lesson. This evaluation tool is simple but focused in its scope as the importance of measuring the content and its ability to meet the objectives is the paramount concern. Some matters to be considered were as follows:
Is the Great Depression an appropriate topic for our target student?
Is the lesson length appropriate?
Is the organization of the content appropriate?
Is Book Builder an appropriate vehicle for the content delivery?
Does the Book Builder lesson meet the stated goals of the project?
A copy of the SME evaluation form used may be downloaded from here
The following changes were made to the Great Depression Book Builder as a result of our formative evaluation:
Added an introductory slide and screencast video explaining the features of Book Builder and our expectations of students (i.e., use the coaches on every page, use the glossary when unfamiliar with a word or term, and access every link)
Changed the voice of the first coach to a female voice which enunciates more clearly
Eliminated text on text-heavy slides
Added interactive components (audio/video links) on slides previously without any links
Changed the names of our coaches to better describe their particular use (Dee Do-You-Know and Relate-It-Ralph)
Sped up page loading by resizing the photos
Distinguished definition links (italicized) from media links (boldfaced)
Reorganized the hierarchy or the material being presented by reordering pagination
Expanded the jazz slide to address the Harlem Renaissance and included it in the glossary.
Deleted the original page 18 Horror Movies part 2 as it was redundant with page 17 Horror Movies part 1
On the Dance Marathon page, added a YouTube clip from the era and deleted the clip to and content about "They Shoot Horses, Don't They?" as a means to keep the project focused on the period rather than later efforts to dramatize the period
A design evaluation will be conducted to verify that all the planned software design features have been incorporated and are functional. This evaluation will be conducted by a person external to the design team via a checklist. The evaluator will verify Alt Text is available for every photograph; coach graphics and voices are implemented; video, audio, and text links are valid; page layout and text formatting are standard on every page, and color choices are complementary.
A content evaluation will verify that the content (text, audio and video) incorporated in the lesson are accurate, appropriate and all sources are sited correctly.
This evaluation will be completed by the SME from The Great Designer’s design team via checklist. The evaluator will verify that the content remains accurate after edits and is appropriately aligned to the defined goal and topic.
The learner evaluation will serve three purposes.
1. Verify that learning has occurred.
Verification that learning has indeed occurred will be accomplished by having students complete online pre-tests and post-tests of the content of the Book Builder lesson. The content questions on both tests will be identical and relate to the content addressed in the Book Builder lesson.
2. Verify that the instructional interface (Book Builder) is understood and easily navigated.
The proof that the instructional interface (Book Builder, coaches, etc.) is understood and easily navigated will be provided via direct observation of learner interaction with the Book Builder lesson. Observations will be input into the existing Learner Observation Survey.
3. Verify that the instruction was engaging for the learner.
Verification that the instruction is engaging for the learner will be based on two collection methods:
a.) Direct observation of the learner during the summative evaluation event with results input into the Learner Observation Survey.
b.) Direct questioning of students' engagement and satisfaction with the Great Depression Book Builder via the Learner Post-Test.
Learner Evaluation Methodology
The summative evaluation will require both a facilitator and an official observer to be in attendance. A total of (5) five 9th -12th grade student learners will participate in this group evaluation. The students will volunteer based on a request for participants via school announcements. The evaluation requires a technology lab/classroom setting. Each participant will have access to a computer specifically dedicated to this evaluation event. The lab/classroom must be equipped with “fast” access to the Internet. Each computer system will include a mouse, color monitor and working head phones.
Learners will have in total 1 hour to complete the lesson and tests.
The introduction of the lesson to the group of learners will be carefully scripted to provide guidance to the participants regarding their responsibilities:
The learners are expected to remain focused on their task.
The learners are expected to try their best and complete the lesson.
The learners are expected to request clarification if necessary
The introduction will also provide guidance related to the basic navigation of the Bookbuilder lesson:
How and why to access the coaches
How and why to access the html, video and audio links
How and why to access the glossary
The appropriate method of replying to student prompts.
A facilitator as well as a formal observer will both note learner behaviors and concur on the results; however only the formal ‘observer’ will record the information into the survey. Learner behavior and comments/questions will be recorded via the Learner Observation Survey utilized in the formative evaluation.
Learners will access the on-line pre-test of current knowledge regarding the lesson content and complete it, saving it for later review. The facilitator will verify that the test is complete by looking at each participants screen. Learners will then access the lesson, reading/listening to the content while invoking all multimedia links. The learner will also complete all student response assignments to the best of their ability. Finaly learners will access the on-line post-test regarding the lesson content and complete it, saving it for later review and evaluation of knowledge gained. The facilitator will verify post-test completion by looking at each participants screen.
Peer Evaluation
Learner Evaluation
Instructional Designer Observations
Subject Matter Evaluation (SME) Evaluation
Modifications Based on Formative Evaluation
Summative Evaluation Methodology
Design Evaluation
Content Evaluation
Summative Learner Evaluation
Formative Evaluation Methodology and Results
The Great Designers advocate using the Dick, Carey and Carey model of formative evaluation. Our product is designed to be self-instructional, therefore the emphasis should be on collecting data, analyzing data, and revising instruction based on the analysis of the data. We will employ the following formative evaluations in our evaluation process: peer evaluations, learner evaluations, instructional designer observations, and subject matter expert evaluations.Top
Peer Evaluation
We asked our instructional design peers to evaluate our Great Depression Book Builder lesson and provide feedback on their experience and impressions through an online survey. Eight students from EDET 722 served as peer evaluators of the design, presentation of information and content of our Book Builder lesson. The survey respondents were consistent and forthcoming in their objective findings, opinions and suggestions.The rating scale for the fourteen individual prompts that constitute the “The Great Depression” formative evaluation is:
All of the input, excepting one, was either Agree or Strongly Agree. The one exception was a “disagree” recorded for the prompt, “The Book Builder lesson makes me want to learn more about the Great Depression.” General responses indicate, “The Great Depression Book Builder site is easy to use. The content of the lesson is written in clear and simple language. The content is easy to understand and follow. The lesson has an attractive presentation I want to see more social studies content presented in this fashion.” This generally positive response to the lesson’s content and delivery method indicates a basis for the continuation of development.
In addition, the peer survey included 1 question which related to the evaluator’s overall impression of the product. Using a scale of Very Good, Good, Average, Poor and Very Poor, respondents answered, “Overall, how do you rate the Great Depression Book Builder lesson?” The Great Designers note with pleasure that 25% rated the Great Depression Book Builder lesson “very good” and 75% rated it “good”.
Two “open-ended” questions offered the opportunity for respondents to identify the components of the lesson that they liked the best as well as the least. Our peers appreciated the audio clips, music, external article(s), glossary and student response section the best. The least liked aspects were related to the “over long” length of the lesson and overabundance of text, lack of audio/narration on certain pages.
The final question requested valued input from our peers regarding suggestions for improvement to our Book Builder lesson. A summation of the suggestions indicated a need to reduce the text significantly on many of our pages, perhaps replacing it with a narrator coach or narration clips. Another peer suggested the inclusion of more of what makes the content interactive; meaning more audio, graphics and video links on every page. The utilization of our Coaches could be improved by programming them to suggest clicking on available audio clips. One reviewer indicated this change would make them more appealing and exciting. And finally was an appeal for the use of more color on the pages in pursuit of a more attractive design.
The three questions that received the lowest rankings were:
This is particularly troubling because our stated goal is to improve learner scores by increasing learner engagement.
In summation, the input from our peers (external instructional designers) was invaluable. This information in combination with the target student observations, subject matter expert opinions and the experience of our Instructional Design team guided the appropriate modifications to our Book Builder Lesson.
Top
Learner Evaluation
Three high school students completed the Great Depression Book Builder lesson and provided feedback on their experience and impressions. Students completed the lesson individually with an instructional designer in attendance and took the same online survey administered to our peer external instructional designers. The students' responses were aggregated with the peer responses in the results reported above.Top
Instructional Designer Observations
Two members of our design team completed observations of a total of three high school students' interaction with the product interface. Observations were recorded related to the level of attention, enthusiasm and reaction to the lesson’s delivery method, content, and student response. Instructional designers used the observation form to complete each observation.Observation Results
Students spent approximately 1 to 5 minutes on each page. Total time to complete the Book Builder ranged from 30 minutes to 1 hour.Generally, students did not access all the interactive links. On most pages, only 33% of the students accessed all the interactive links, and students gave the audio and video clips no attention. Students often questioned if they had to watch an entire clip. Students commented that the NPR story was too long and did not listen to the entire story. One student did not even open the story. The students did not use the glossary at all.
Students were inconsistent in launching the coaches. The voice of Hali was difficult to understand without reading along, and students found the coaches to be too slow and too immature for the high school population. Overall the coaches seemed to be a distraction to the students.
Students were also inconsistent in using the student response area. On most pages, 66%-100% of students used the student response area. Students asked if they really had to answer all the questions and expressed frustration that the student response area was small and difficult to edit. On some pages, students asked for help in answering the questions (e.g., Do you know the unemployment rate? What movie stars would I know from the 1930s?).
Overall, students seemed to be at least moderately engaged with the lesson although the slow speed of the coaches and the pages was a distraction. The content was age- and grade-appropriate. The questions were challenging and required the students to put forth effort and thought. On some occasions, students seemed to make connections between the lesson content and prior learning or personal experiences. One student did not seem to put forth much effort and did not access many links. He attributed this lack of effort to not wanting to wait on the school’s slow internet and said that he would access the links if he were on a faster internet connection.
Top
Subject Matter Expert (SME) Evaluation
The subject matter expert evaluation sought to answer whether or not the content selected is appropriate given the objectives chosen by the team for the Great Depression Book Builder lesson. This evaluation tool is simple but focused in its scope as the importance of measuring the content and its ability to meet the objectives is the paramount concern. Some matters to be considered were as follows:A copy of the SME evaluation form used may be downloaded from here
The results of the SME evaluation can be viewed or downloaded here
Top
Modifications Based on Formative Evaluation
The following changes were made to the Great Depression Book Builder as a result of our formative evaluation:Top
Summative Evaluation Methodology
Design Evaluation
A design evaluation will be conducted to verify that all the planned software design features have been incorporated and are functional. This evaluation will be conducted by a person external to the design team via a checklist. The evaluator will verify Alt Text is available for every photograph; coach graphics and voices are implemented; video, audio, and text links are valid; page layout and text formatting are standard on every page, and color choices are complementary.Top
Content Evaluation
A content evaluation will verify that the content (text, audio and video) incorporated in the lesson are accurate, appropriate and all sources are sited correctly.This evaluation will be completed by the SME from The Great Designer’s design team via checklist. The evaluator will verify that the content remains accurate after edits and is appropriately aligned to the defined goal and topic.
Top
Learner Evaluation
The learner evaluation will serve three purposes.1. Verify that learning has occurred.
Verification that learning has indeed occurred will be accomplished by having students complete online pre-tests and post-tests of the content of the Book Builder lesson. The content questions on both tests will be identical and relate to the content addressed in the Book Builder lesson.
2. Verify that the instructional interface (Book Builder) is understood and easily navigated.
The proof that the instructional interface (Book Builder, coaches, etc.) is understood and easily navigated will be provided via direct observation of learner interaction with the Book Builder lesson. Observations will be input into the existing Learner Observation Survey.
3. Verify that the instruction was engaging for the learner.
Verification that the instruction is engaging for the learner will be based on two collection methods:
a.) Direct observation of the learner during the summative evaluation event with results input into the Learner Observation Survey.
b.) Direct questioning of students' engagement and satisfaction with the Great Depression Book Builder via the Learner Post-Test.
Learner Evaluation Methodology
The summative evaluation will require both a facilitator and an official observer to be in attendance. A total of (5) five 9th -12th grade student learners will participate in this group evaluation. The students will volunteer based on a request for participants via school announcements. The evaluation requires a technology lab/classroom setting. Each participant will have access to a computer specifically dedicated to this evaluation event. The lab/classroom must be equipped with “fast” access to the Internet. Each computer system will include a mouse, color monitor and working head phones.Learners will have in total 1 hour to complete the lesson and tests.
The introduction of the lesson to the group of learners will be carefully scripted to provide guidance to the participants regarding their responsibilities:
The introduction will also provide guidance related to the basic navigation of the Bookbuilder lesson:
A facilitator as well as a formal observer will both note learner behaviors and concur on the results; however only the formal ‘observer’ will record the information into the survey. Learner behavior and comments/questions will be recorded via the Learner Observation Survey utilized in the formative evaluation.
Learners will access the on-line pre-test of current knowledge regarding the lesson content and complete it, saving it for later review. The facilitator will verify that the test is complete by looking at each participants screen. Learners will then access the lesson, reading/listening to the content while invoking all multimedia links. The learner will also complete all student response assignments to the best of their ability. Finaly learners will access the on-line post-test regarding the lesson content and complete it, saving it for later review and evaluation of knowledge gained. The facilitator will verify post-test completion by looking at each participants screen.
Top